Problem Statement¶

Business Context¶

Workplace safety in hazardous environments like construction sites and industrial plants is crucial to prevent accidents and injuries. One of the most important safety measures is ensuring workers wear safety helmets, which protect against head injuries from falling objects and machinery. Non-compliance with helmet regulations increases the risk of serious injuries or fatalities, making effective monitoring essential, especially in large-scale operations where manual oversight is prone to errors and inefficiency.

To overcome these challenges, SafeGuard Corp plans to develop an automated image analysis system capable of detecting whether workers are wearing safety helmets. This system will improve safety enforcement, ensuring compliance and reducing the risk of head injuries. By automating helmet monitoring, SafeGuard aims to enhance efficiency, scalability, and accuracy, ultimately fostering a safer work environment while minimizing human error in safety oversight.

Objective¶

As a data scientist at SafeGuard Corp, you are tasked with developing an image classification model that classifies images into one of two categories:

  • With Helmet: Workers wearing safety helmets.
  • Without Helmet: Workers not wearing safety helmets.

Data Description¶

The dataset consists of 631 images, equally divided into two categories:

  • With Helmet: 311 images showing workers wearing helmets.
  • Without Helmet: 320 images showing workers not wearing helmets.

Dataset Characteristics:

  • Variations in Conditions: Images include diverse environments such as construction sites, factories, and industrial settings, with variations in lighting, angles, and worker postures to simulate real-world conditions.
  • Worker Activities: Workers are depicted in different actions such as standing, using tools, or moving, ensuring robust model learning for various scenarios.

Installing and Importing the Necessary Libraries¶

In [4]:
!pip install tensorflow[and-cuda] numpy==1.25.2 -q

  Installing build dependencies ... done
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
In [1]:
import tensorflow as tf
print("Num GPUs Available:", len(tf.config.list_physical_devices('GPU')))
print(tf.__version__)

Num GPUs Available: 1
2.19.0

Note:

  • After running the above cell, kindly restart the notebook kernel (for Jupyter Notebook) or runtime (for Google Colab) and run all cells sequentially from the next cell.

  • On executing the above line of code, you might see a warning regarding package dependencies. This error message can be ignored as the above code ensures that all necessary libraries and their dependencies are maintained to successfully execute the code in this notebook.

In [3]:
import os
import random
import numpy as np                                                                               # Importing numpy for Matrix Operations
import pandas as pd
import seaborn as sns
import matplotlib.image as mpimg                                                                              # Importing pandas to read CSV files
import matplotlib.pyplot as plt                                                                  # Importting matplotlib for Plotting and visualizing images
import math                                                                                      # Importing math module to perform mathematical operations
import cv2


# Tensorflow modules
import keras
import tensorflow as tf
from tensorflow.keras.preprocessing.image import ImageDataGenerator                              # Importing the ImageDataGenerator for data augmentation
from tensorflow.keras.models import Sequential                                                   # Importing the sequential module to define a sequential model
from tensorflow.keras.layers import Dense,Dropout,Flatten,Conv2D,MaxPooling2D,BatchNormalization # Defining all the layers to build our CNN Model
from tensorflow.keras.optimizers import Adam,SGD                                                 # Importing the optimizers which can be used in our model
from sklearn import preprocessing                                                                # Importing the preprocessing module to preprocess the data
from sklearn.model_selection import train_test_split                                             # Importing train_test_split function to split the data into train and test
from sklearn.metrics import confusion_matrix
from tensorflow.keras.models import Model
from keras.applications.vgg16 import VGG16                                               # Importing confusion_matrix to plot the confusion matrix

# Display images using OpenCV
from google.colab.patches import cv2_imshow

#Imports functions for evaluating the performance of machine learning models
from sklearn.metrics import confusion_matrix, f1_score,accuracy_score, recall_score, precision_score, classification_report
from sklearn.metrics import mean_squared_error as mse                                                 # Importing cv2_imshow from google.patches to display images

# Ignore warnings
import warnings
warnings.filterwarnings('ignore')
In [4]:
# Set the seed using keras.utils.set_random_seed. This will set:
# 1) `numpy` seed
# 2) backend random seed
# 3) `python` random seed
tf.keras.utils.set_random_seed(812)

Data Overview¶

Loading the data¶

In [7]:
from google.colab import drive
drive.mount('/content/drive')

Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
In [11]:
images = np.load('/content/drive/MyDrive/Colab Notebooks/helmet/data/images_proj.npy')
labels_df = pd.read_csv('/content/drive/MyDrive/Colab Notebooks/helmet/data/Labels_proj.csv')

print("Dataset Information:")
print(f"Images shape: {images.shape}")
print(f"Images dtype: {images.dtype}")
print(f"Images min value: {images.min()}, max value: {images.max()}")
print(f"Number of labels: {len(labels_df)}")
print(f"Labels shape: {labels_df.shape}")

# Display first few labels
print("\nFirst 10 labels:")
print(labels_df.head(10))

# Convert labels to numpy array for easier handling
labels = labels_df['Label'].values
print(f"\nUnique labels: {np.unique(labels)}")
print(f"Label distribution:")
print(f"Class 0 (Without Helmet): {np.sum(labels == 0)} images")
print(f"Class 1 (With Helmet): {np.sum(labels == 1)} images")

Dataset Information:
Images shape: (631, 200, 200, 3)
Images dtype: uint8
Images min value: 0, max value: 255
Number of labels: 631
Labels shape: (631, 1)

First 10 labels:
   Label
0      1
1      1
2      1
3      1
4      1
5      1
6      1
7      1
8      1
9      1

Unique labels: [0 1]
Label distribution:
Class 0 (Without Helmet): 320 images
Class 1 (With Helmet): 311 images

Exploratory Data Analysis¶

Plot random images from each of the classes and print their corresponding labels.¶

In [12]:
def plot_sample_images(images, labels, n_samples=8):
    """
    Plot random sample images from each class
    """
    fig, axes = plt.subplots(2, n_samples, figsize=(16, 6))

    # Get indices for each class
    class_0_indices = np.where(labels == 0)[0]
    class_1_indices = np.where(labels == 1)[0]

    # Randomly sample indices
    random_class_0 = np.random.choice(class_0_indices, n_samples, replace=False)
    random_class_1 = np.random.choice(class_1_indices, n_samples, replace=False)

    # Plot class 0 (Without Helmet)
    for i, idx in enumerate(random_class_0):
        axes[0, i].imshow(images[idx])
        axes[0, i].axis('off')
        if i == 0:
            axes[0, i].set_title('Without Helmet', fontsize=12, fontweight='bold')
        axes[0, i].set_title(f'Index: {idx}', fontsize=8)

    # Plot class 1 (With Helmet)
    for i, idx in enumerate(random_class_1):
        axes[1, i].imshow(images[idx])
        axes[1, i].axis('off')
        if i == 0:
            axes[1, i].set_title('With Helmet', fontsize=12, fontweight='bold')
        axes[1, i].set_title(f'Index: {idx}', fontsize=8)

    plt.suptitle('Sample Images from Each Class', fontsize=14, fontweight='bold')
    plt.tight_layout()
    plt.show()

# Plot sample images
plot_sample_images(images, labels, n_samples=8)
No description has been provided for this image

Checking for class imbalance¶

In [13]:
# Visualize class distribution
plt.figure(figsize=(12, 5))

# Bar plot
plt.subplot(1, 2, 1)
class_counts = pd.Series(labels).value_counts().sort_index()
bars = plt.bar(['Without Helmet (0)', 'With Helmet (1)'], class_counts.values,
               color=['#ff6b6b', '#4ecdc4'])
plt.ylabel('Number of Images')
plt.title('Class Distribution')
for bar, count in zip(bars, class_counts.values):
    plt.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 5,
             str(count), ha='center', va='bottom', fontweight='bold')

# Pie chart
plt.subplot(1, 2, 2)
plt.pie(class_counts.values, labels=['Without Helmet', 'With Helmet'],
        autopct='%1.1f%%', colors=['#ff6b6b', '#4ecdc4'], startangle=90)
plt.title('Class Distribution (Percentage)')

plt.suptitle('Dataset Class Balance Analysis', fontsize=14, fontweight='bold')
plt.tight_layout()
plt.show()

# Print imbalance ratio
imbalance_ratio = class_counts[0] / class_counts[1]
print(f"Class imbalance ratio (Without/With): {imbalance_ratio:.2f}")
print(f"The dataset is {'balanced' if 0.9 <= imbalance_ratio <= 1.1 else 'slightly imbalanced'}")
No description has been provided for this image 
Class imbalance ratio (Without/With): 1.03
The dataset is balanced

Data Preprocessing¶

Converting images to grayscale¶

In [14]:
def convert_to_grayscale(images):
    """
    Convert RGB images to grayscale using weighted average method
    """
    # Create array to store grayscale images
    grayscale_images = np.zeros((images.shape[0], images.shape[1], images.shape[2], 1),
                                dtype=images.dtype)

    # Convert each image to grayscale
    for i in range(images.shape[0]):
        # Use cv2 for proper grayscale conversion
        gray = cv2.cvtColor(images[i], cv2.COLOR_RGB2GRAY)
        grayscale_images[i] = gray.reshape(images.shape[1], images.shape[2], 1)

    return grayscale_images

# Convert images to grayscale
grayscale_images = convert_to_grayscale(images)
print(f"Original images shape: {images.shape}")
print(f"Grayscale images shape: {grayscale_images.shape}")

# Visualize conversion
fig, axes = plt.subplots(2, 4, figsize=(12, 6))
sample_indices = np.random.choice(len(images), 4, replace=False)

for i, idx in enumerate(sample_indices):
    # Original images
    axes[0, i].imshow(images[idx])
    axes[0, i].axis('off')
    axes[0, i].set_title(f'Original (Index: {idx})')

    # Grayscale images
    axes[1, i].imshow(grayscale_images[idx].squeeze(), cmap='gray')
    axes[1, i].axis('off')
    axes[1, i].set_title(f'Grayscale (Label: {labels[idx]})')

plt.suptitle('RGB to Grayscale Conversion', fontsize=14, fontweight='bold')
plt.tight_layout()
plt.show()

Original images shape: (631, 200, 200, 3)
Grayscale images shape: (631, 200, 200, 1)
No description has been provided for this image

Splitting the dataset¶

In [15]:
from sklearn.model_selection import train_test_split

# First split: 80% train+val, 20% test
X_temp, X_test, y_temp, y_test = train_test_split(
    grayscale_images, labels,
    test_size=0.2,
    random_state=42,
    stratify=labels
)

# Second split: From the 80%, take 75% for train (60% of total) and 25% for val (20% of total)
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp,
    test_size=0.25,
    random_state=42,
    stratify=y_temp
)

# Convert labels to pandas Series for consistency
y_train = pd.Series(y_train)
y_val = pd.Series(y_val)
y_test = pd.Series(y_test)

print("Dataset split:")
print(f"Training set: {X_train.shape[0]} images ({X_train.shape[0]/len(grayscale_images)*100:.1f}%)")
print(f"Validation set: {X_val.shape[0]} images ({X_val.shape[0]/len(grayscale_images)*100:.1f}%)")
print(f"Test set: {X_test.shape[0]} images ({X_test.shape[0]/len(grayscale_images)*100:.1f}%)")
print(f"\nClass distribution in splits:")
print(f"Train - Class 0: {(y_train==0).sum()}, Class 1: {(y_train==1).sum()}")
print(f"Val - Class 0: {(y_val==0).sum()}, Class 1: {(y_val==1).sum()}")
print(f"Test - Class 0: {(y_test==0).sum()}, Class 1: {(y_test==1).sum()}")

Dataset split:
Training set: 378 images (59.9%)
Validation set: 126 images (20.0%)
Test set: 127 images (20.1%)

Class distribution in splits:
Train - Class 0: 192, Class 1: 186
Val - Class 0: 64, Class 1: 62
Test - Class 0: 64, Class 1: 63

Data Normalization¶

In [16]:
X_train = X_train.astype('float32') / 255.0
X_val = X_val.astype('float32') / 255.0
X_test = X_test.astype('float32') / 255.0

print("Data normalization completed:")
print(f"X_train min/max: {X_train.min():.3f}/{X_train.max():.3f}")
print(f"X_val min/max: {X_val.min():.3f}/{X_val.max():.3f}")
print(f"X_test min/max: {X_test.min():.3f}/{X_test.max():.3f}")

# Verify shapes
print(f"\nFinal shapes:")
print(f"X_train: {X_train.shape}, y_train: {y_train.shape}")
print(f"X_val: {X_val.shape}, y_val: {y_val.shape}")
print(f"X_test: {X_test.shape}, y_test: {y_test.shape}")

Data normalization completed:
X_train min/max: 0.000/1.000
X_val min/max: 0.000/1.000
X_test min/max: 0.000/1.000

Final shapes:
X_train: (378, 200, 200, 1), y_train: (378,)
X_val: (126, 200, 200, 1), y_val: (126,)
X_test: (127, 200, 200, 1), y_test: (127,)

Model Building¶

Model Evaluation Criterion¶

Utility Functions¶

In [31]:
# defining a function to compute different metrics to check performance of a classification model built using statsmodels
# def model_performance_classification(model, predictors, target):
#     """
#     Function to compute different metrics to check classification model performance

#     model: classifier
#     predictors: independent variables (image data)
#     target: dependent variable
#     """

#     # checking which probabilities are greater than threshold
#     pred = model.predict(predictors).reshape(-1) > 0.5

#     target = target.to_numpy().reshape(-1)


#     acc = accuracy_score(target, pred)  # to compute Accuracy
#     recall = recall_score(target, pred, average='weighted')  # to compute Recall
#     precision = precision_score(target, pred, average='weighted')  # to compute Precision
#     f1 = f1_score(target, pred, average='weighted')  # to compute F1-score

#     # creating a dataframe of metrics
#     df_perf = pd.DataFrame({"Accuracy": acc, "Recall": recall, "Precision": precision, "F1 Score": f1,},index=[0],)

#     return df_perf

def model_performance_classification(model, predictors, target):
    """
    Function to compute different metrics to check classification model performance

    model: classifier
    predictors: independent variables
    target: dependent variable
    """

    # checking which probabilities are greater than threshold
    pred = model.predict(predictors).reshape(-1)>0.5

    target = target.to_numpy().reshape(-1)


    acc = accuracy_score(target, pred)  # to compute Accuracy
    recall = recall_score(target, pred, average='weighted')  # to compute Recall
    precision = precision_score(target, pred, average='weighted')  # to compute Precision
    f1 = f1_score(target, pred, average='weighted')  # to compute F1-score

    # creating a dataframe of metrics
    df_perf = pd.DataFrame({"Accuracy": acc, "Recall": recall, "Precision": precision, "F1 Score": f1,},index=[0],)

    return df_perf
In [30]:
def plot_confusion_matrix(model,predictors,target,ml=False):
    """
    Function to plot the confusion matrix

    model: classifier
    predictors: independent variables
    target: dependent variable
    ml: To specify if the model used is an sklearn ML model or not (True means ML model)
    """

    # checking which probabilities are greater than threshold
    pred = model.predict(predictors).reshape(-1)>0.5

    target = target.to_numpy().reshape(-1)

    # Plotting the Confusion Matrix using confusion matrix() function which is also predefined tensorflow module
    confusion_matrix = tf.math.confusion_matrix(target,pred)
    f, ax = plt.subplots(figsize=(10, 8))
    sns.heatmap(
        confusion_matrix,
        annot=True,
        linewidths=.4,
        fmt="d",
        square=True,
        ax=ax
    )
    plt.show()

Model 1: Simple Convolutional Neural Network (CNN)¶

In [23]:
def build_simple_cnn(input_shape):
    """
    Build a simple CNN architecture for helmet detection
    """
    model = Sequential([
        # First Convolutional Block
        Conv2D(32, (3, 3), activation='relu', padding='same', input_shape=input_shape),
        BatchNormalization(),
        Conv2D(32, (3, 3), activation='relu', padding='same'),
        BatchNormalization(),
        MaxPooling2D((2, 2)),
        Dropout(0.25),

        # Second Convolutional Block
        Conv2D(64, (3, 3), activation='relu', padding='same'),
        BatchNormalization(),
        Conv2D(64, (3, 3), activation='relu', padding='same'),
        BatchNormalization(),
        MaxPooling2D((2, 2)),
        Dropout(0.25),

        # Third Convolutional Block
        Conv2D(128, (3, 3), activation='relu', padding='same'),
        BatchNormalization(),
        Conv2D(128, (3, 3), activation='relu', padding='same'),
        BatchNormalization(),
        MaxPooling2D((2, 2)),
        Dropout(0.4),

        # Fully Connected Layers
        Flatten(),
        Dense(256, activation='relu'),
        BatchNormalization(),
        Dropout(0.5),
        Dense(128, activation='relu'),
        BatchNormalization(),
        Dropout(0.5),
        Dense(1, activation='sigmoid')  # Binary classification
    ])

    return model

# Build and compile the model
model1 = build_simple_cnn(input_shape=(200, 200, 1))

# Compile with appropriate settings
model1.compile(
    optimizer=Adam(learning_rate=0.001),
    loss='binary_crossentropy',
    metrics=['accuracy', tf.keras.metrics.Precision(), tf.keras.metrics.Recall()]
)

# Display model architecture
print("Model 1: Simple CNN Architecture")
model1.summary()

# Train the model
history1 = model1.fit(
    X_train, y_train,
    validation_data=(X_val, y_val),
    epochs=30,
    batch_size=32,
    verbose=1,
    callbacks=[
        tf.keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True),
        tf.keras.callbacks.ReduceLROnPlateau(patience=3, factor=0.5, min_lr=1e-6)
    ]
)

# Evaluate Model 1
print("\n" + "="*50)
print("Model 1 Performance Evaluation")
print("="*50)
train_perf1 = model_performance_classification(model1, X_train, y_train)
val_perf1 = model_performance_classification(model1, X_val, y_val)
print("Training Performance:")
print(train_perf1)
print("\nValidation Performance:")
print(val_perf1)

Model 1: Simple CNN Architecture

Model: "sequential_2"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ conv2d_12 (Conv2D)              │ (None, 200, 200, 32)   │           320 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization_16          │ (None, 200, 200, 32)   │           128 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_13 (Conv2D)              │ (None, 200, 200, 32)   │         9,248 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization_17          │ (None, 200, 200, 32)   │           128 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ max_pooling2d_6 (MaxPooling2D)  │ (None, 100, 100, 32)   │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_10 (Dropout)            │ (None, 100, 100, 32)   │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_14 (Conv2D)              │ (None, 100, 100, 64)   │        18,496 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization_18          │ (None, 100, 100, 64)   │           256 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_15 (Conv2D)              │ (None, 100, 100, 64)   │        36,928 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization_19          │ (None, 100, 100, 64)   │           256 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ max_pooling2d_7 (MaxPooling2D)  │ (None, 50, 50, 64)     │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_11 (Dropout)            │ (None, 50, 50, 64)     │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_16 (Conv2D)              │ (None, 50, 50, 128)    │        73,856 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization_20          │ (None, 50, 50, 128)    │           512 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_17 (Conv2D)              │ (None, 50, 50, 128)    │       147,584 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization_21          │ (None, 50, 50, 128)    │           512 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ max_pooling2d_8 (MaxPooling2D)  │ (None, 25, 25, 128)    │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_12 (Dropout)            │ (None, 25, 25, 128)    │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ flatten_2 (Flatten)             │ (None, 80000)          │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_6 (Dense)                 │ (None, 256)            │    20,480,256 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization_22          │ (None, 256)            │         1,024 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_13 (Dropout)            │ (None, 256)            │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_7 (Dense)                 │ (None, 128)            │        32,896 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization_23          │ (None, 128)            │           512 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_14 (Dropout)            │ (None, 128)            │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_8 (Dense)                 │ (None, 1)              │           129 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 20,803,041 (79.36 MB)

 Trainable params: 20,801,377 (79.35 MB)

 Non-trainable params: 1,664 (6.50 KB)

Epoch 1/30
12/12 ━━━━━━━━━━━━━━━━━━━━ 27s 1s/step - accuracy: 0.8050 - loss: 0.4329 - precision_2: 0.7927 - recall_2: 0.8480 - val_accuracy: 0.5079 - val_loss: 2.9206 - val_precision_2: 0.0000e+00 - val_recall_2: 0.0000e+00 - learning_rate: 0.0010
Epoch 2/30
12/12 ━━━━━━━━━━━━━━━━━━━━ 21s 128ms/step - accuracy: 0.9753 - loss: 0.1011 - precision_2: 0.9948 - recall_2: 0.9575 - val_accuracy: 0.5079 - val_loss: 6.0638 - val_precision_2: 0.0000e+00 - val_recall_2: 0.0000e+00 - learning_rate: 0.0010
Epoch 3/30
12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 126ms/step - accuracy: 0.9855 - loss: 0.0381 - precision_2: 0.9987 - recall_2: 0.9734 - val_accuracy: 0.5079 - val_loss: 5.9870 - val_precision_2: 0.0000e+00 - val_recall_2: 0.0000e+00 - learning_rate: 0.0010
Epoch 4/30
12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 126ms/step - accuracy: 0.9908 - loss: 0.0428 - precision_2: 0.9941 - recall_2: 0.9883 - val_accuracy: 0.5079 - val_loss: 3.6192 - val_precision_2: 0.0000e+00 - val_recall_2: 0.0000e+00 - learning_rate: 0.0010
Epoch 5/30
12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 131ms/step - accuracy: 0.9919 - loss: 0.0238 - precision_2: 0.9949 - recall_2: 0.9896 - val_accuracy: 0.5079 - val_loss: 4.3336 - val_precision_2: 0.0000e+00 - val_recall_2: 0.0000e+00 - learning_rate: 5.0000e-04
Epoch 6/30
12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 126ms/step - accuracy: 1.0000 - loss: 0.0131 - precision_2: 1.0000 - recall_2: 1.0000 - val_accuracy: 0.5079 - val_loss: 4.7845 - val_precision_2: 0.0000e+00 - val_recall_2: 0.0000e+00 - learning_rate: 5.0000e-04

==================================================
Model 1 Performance Evaluation
==================================================
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 250ms/step
4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 189ms/step
Training Performance:
   Accuracy    Recall  Precision  F1 Score
0  0.507937  0.507937   0.257999  0.342189

Validation Performance:
   Accuracy    Recall  Precision  F1 Score
0  0.507937  0.507937   0.257999  0.342189

Vizualizing the predictions¶

In [26]:
def visualize_predictions(model, X_data, y_true, model_name="Model", n_samples=8):
    """
    Visualize model predictions on sample images
    """
    # Get predictions
    predictions = model.predict(X_data[:n_samples])
    pred_classes = (predictions > 0.5).astype(int).reshape(-1)

    # Plot
    fig, axes = plt.subplots(2, 4, figsize=(12, 6))
    axes = axes.ravel()

    for i in range(min(n_samples, 8)):
        axes[i].imshow(X_data[i].squeeze(), cmap='gray')
        axes[i].axis('off')

        true_label = "With Helmet" if y_true.iloc[i] == 1 else "Without Helmet"
        pred_label = "With Helmet" if pred_classes[i] == 1 else "Without Helmet"
        confidence = predictions[i][0] if pred_classes[i] == 1 else 1 - predictions[i][0]

        color = 'green' if y_true.iloc[i] == pred_classes[i] else 'red'
        axes[i].set_title(f'True: {true_label}\nPred: {pred_label} ({confidence:.2%})',
                         color=color, fontsize=9)

    plt.suptitle(f'{model_name} - Sample Predictions', fontsize=14, fontweight='bold')
    plt.tight_layout()
    plt.show()

# Visualize predictions for Model 1
visualize_predictions(model1, X_val, y_val, "Simple CNN")
plot_confusion_matrix(model1, X_val, y_val)

1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
No description has been provided for this image 
4/4 ━━━━━━━━━━━━━━━━━━━━ 0s 46ms/step
No description has been provided for this image

Model 2: (VGG-16 (Base))¶

In [34]:
X_train_rgb = np.repeat(X_train, 3, axis=-1)
X_val_rgb = np.repeat(X_val, 3, axis=-1)
X_test_rgb = np.repeat(X_test, 3, axis=-1)

# Build VGG16 base model
def build_vgg16_base(input_shape):
    """
    Build VGG16 model with frozen base layers
    """
    # Load pre-trained VGG16
    base_model = VGG16(
        weights='imagenet',
        include_top=False,
        input_shape=input_shape,
        pooling='avg'
    )

    # Freeze base model layers
    base_model.trainable = False

    # Build model
    model = Sequential([
        base_model,
        Dense(1, activation='sigmoid')
    ])

    return model

# Build and compile Model 2
model2 = build_vgg16_base(input_shape=(200, 200, 3))

model2.compile(
    optimizer=Adam(learning_rate=0.001),
    loss='binary_crossentropy',
    metrics=['accuracy', tf.keras.metrics.Precision(), tf.keras.metrics.Recall()]
)

print("Model 2: VGG16 Base Architecture")
model2.summary()

# Train Model 2
history2 = model2.fit(
    X_train_rgb, y_train,
    validation_data=(X_val_rgb, y_val),
    epochs=20,
    batch_size=32,
    verbose=1,
    callbacks=[
        tf.keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True),
        tf.keras.callbacks.ReduceLROnPlateau(patience=3, factor=0.5, min_lr=1e-6)
    ]
)

# Evaluate Model 2
print("\n" + "="*50)
print("Model 2 Performance Evaluation")
print("="*50)
train_perf2 = model_performance_classification(model2, X_train_rgb, y_train)
val_perf2 = model_performance_classification(model2, X_val_rgb, y_val)
print("Training Performance:")
print(train_perf2)
print("\nValidation Performance:")
print(val_perf2)

Model 2: VGG16 Base Architecture

Model: "sequential_7"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ vgg16 (Functional)              │ (None, 512)            │    14,714,688 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_13 (Dense)                │ (None, 1)              │           513 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 14,715,201 (56.13 MB)

 Trainable params: 513 (2.00 KB)

 Non-trainable params: 14,714,688 (56.13 MB)

Epoch 1/20
12/12 ━━━━━━━━━━━━━━━━━━━━ 9s 571ms/step - accuracy: 0.5335 - loss: 0.6916 - precision_7: 0.5398 - recall_7: 0.4277 - val_accuracy: 0.7460 - val_loss: 0.6298 - val_precision_7: 0.6630 - val_recall_7: 0.9839 - learning_rate: 0.0010
Epoch 2/20
12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 205ms/step - accuracy: 0.8451 - loss: 0.6099 - precision_7: 0.7962 - recall_7: 0.9384 - val_accuracy: 0.9762 - val_loss: 0.5558 - val_precision_7: 0.9538 - val_recall_7: 1.0000 - learning_rate: 0.0010
Epoch 3/20
12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 202ms/step - accuracy: 0.9810 - loss: 0.5427 - precision_7: 0.9846 - recall_7: 0.9781 - val_accuracy: 1.0000 - val_loss: 0.4912 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010
Epoch 4/20
12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 202ms/step - accuracy: 0.9967 - loss: 0.4840 - precision_7: 1.0000 - recall_7: 0.9938 - val_accuracy: 1.0000 - val_loss: 0.4360 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010
Epoch 5/20
12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 210ms/step - accuracy: 0.9967 - loss: 0.4327 - precision_7: 1.0000 - recall_7: 0.9938 - val_accuracy: 1.0000 - val_loss: 0.3889 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010
Epoch 6/20
12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 205ms/step - accuracy: 0.9967 - loss: 0.3885 - precision_7: 1.0000 - recall_7: 0.9938 - val_accuracy: 1.0000 - val_loss: 0.3486 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010
Epoch 7/20
12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 203ms/step - accuracy: 0.9967 - loss: 0.3505 - precision_7: 1.0000 - recall_7: 0.9938 - val_accuracy: 1.0000 - val_loss: 0.3139 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010
Epoch 8/20
12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 204ms/step - accuracy: 0.9967 - loss: 0.3179 - precision_7: 1.0000 - recall_7: 0.9938 - val_accuracy: 1.0000 - val_loss: 0.2842 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010
Epoch 9/20
12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 205ms/step - accuracy: 0.9967 - loss: 0.2896 - precision_7: 1.0000 - recall_7: 0.9938 - val_accuracy: 1.0000 - val_loss: 0.2586 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010
Epoch 10/20
12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 212ms/step - accuracy: 0.9967 - loss: 0.2651 - precision_7: 1.0000 - recall_7: 0.9938 - val_accuracy: 1.0000 - val_loss: 0.2364 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010
Epoch 11/20
12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 210ms/step - accuracy: 1.0000 - loss: 0.2436 - precision_7: 1.0000 - recall_7: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.2170 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010
Epoch 12/20
12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 207ms/step - accuracy: 1.0000 - loss: 0.2248 - precision_7: 1.0000 - recall_7: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.2001 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010
Epoch 13/20
12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 208ms/step - accuracy: 1.0000 - loss: 0.2083 - precision_7: 1.0000 - recall_7: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1852 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010
Epoch 14/20
12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 207ms/step - accuracy: 1.0000 - loss: 0.1936 - precision_7: 1.0000 - recall_7: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1720 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010
Epoch 15/20
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 296ms/step - accuracy: 1.0000 - loss: 0.1806 - precision_7: 1.0000 - recall_7: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1602 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010
Epoch 16/20
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 225ms/step - accuracy: 1.0000 - loss: 0.1689 - precision_7: 1.0000 - recall_7: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1498 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010
Epoch 17/20
12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 238ms/step - accuracy: 1.0000 - loss: 0.1584 - precision_7: 1.0000 - recall_7: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1404 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010
Epoch 18/20
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 248ms/step - accuracy: 1.0000 - loss: 0.1490 - precision_7: 1.0000 - recall_7: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1319 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010
Epoch 19/20
12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 226ms/step - accuracy: 1.0000 - loss: 0.1404 - precision_7: 1.0000 - recall_7: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1243 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010
Epoch 20/20
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 209ms/step - accuracy: 1.0000 - loss: 0.1327 - precision_7: 1.0000 - recall_7: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1174 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010

==================================================
Model 2 Performance Evaluation
==================================================
12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 229ms/step
4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 281ms/step
Training Performance:
   Accuracy  Recall  Precision  F1 Score
0       1.0     1.0        1.0       1.0

Validation Performance:
   Accuracy  Recall  Precision  F1 Score
0       1.0     1.0        1.0       1.0

Visualizing the prediction:¶

In [38]:
visualize_predictions(model2, X_val_rgb, y_val, "VGG16 Base")
plot_confusion_matrix(model2, X_val_rgb, y_val)

1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 194ms/step
No description has been provided for this image 
4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 159ms/step
No description has been provided for this image

Model 3: (VGG-16 (Base + FFNN))¶

In [40]:
def build_vgg16_ffnn(input_shape):
    """
    Build VGG16 model with additional fully connected layers
    """
    # Load pre-trained VGG16
    base_model = VGG16(
        weights='imagenet',
        include_top=False,
        input_shape=input_shape,
        pooling='avg'
    )

    # Freeze base model layers initially
    base_model.trainable = False

    # Build model with additional layers
    model = Sequential([
        base_model,
        Dense(512, activation='relu'),
        BatchNormalization(),
        Dropout(0.5),
        Dense(256, activation='relu'),
        BatchNormalization(),
        Dropout(0.5),
        Dense(128, activation='relu'),
        BatchNormalization(),
        Dropout(0.3),
        Dense(1, activation='sigmoid')
    ])

    return model

# Build and compile Model 3
model3 = build_vgg16_ffnn(input_shape=(200, 200, 3))

model3.compile(
    optimizer=Adam(learning_rate=0.001),
    loss='binary_crossentropy',
    metrics=['accuracy', tf.keras.metrics.Precision(), tf.keras.metrics.Recall()]
)

print("Model 3: VGG16 with FFNN Architecture")
model3.summary()

# Train Model 3 - Phase 1: With frozen base
history3_phase1 = model3.fit(
    X_train_rgb, y_train,
    validation_data=(X_val_rgb, y_val),
    epochs=15,
    batch_size=32,
    verbose=1,
    callbacks=[
        tf.keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True),
        tf.keras.callbacks.ReduceLROnPlateau(patience=3, factor=0.5, min_lr=1e-6)
    ]
)

# Fine-tune: Unfreeze last few layers of base model
model3.layers[0].trainable = True
# Freeze all layers except last 4
for layer in model3.layers[0].layers[:-4]:
    layer.trainable = False

# Recompile with lower learning rate
model3.compile(
    optimizer=Adam(learning_rate=0.0001),
    loss='binary_crossentropy',
    metrics=['accuracy', tf.keras.metrics.Precision(), tf.keras.metrics.Recall()]
)

# Train Model 3 - Phase 2: Fine-tuning
history3_phase2 = model3.fit(
    X_train_rgb, y_train,
    validation_data=(X_val_rgb, y_val),
    epochs=10,
    batch_size=32,
    verbose=1,
    callbacks=[
        tf.keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True),
        tf.keras.callbacks.ReduceLROnPlateau(patience=3, factor=0.5, min_lr=1e-7)
    ]
)

# Evaluate Model 3
print("\n" + "="*50)
print("Model 3 Performance Evaluation")
print("="*50)
train_perf3 = model_performance_classification(model3, X_train_rgb, y_train)
val_perf3 = model_performance_classification(model3, X_val_rgb, y_val)
print("Training Performance:")
print(train_perf3)
print("\nValidation Performance:")
print(val_perf3)

Model 3: VGG16 with FFNN Architecture

Model: "sequential_9"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ vgg16 (Functional)              │ (None, 512)            │    14,714,688 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_18 (Dense)                │ (None, 512)            │       262,656 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization_27          │ (None, 512)            │         2,048 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_18 (Dropout)            │ (None, 512)            │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_19 (Dense)                │ (None, 256)            │       131,328 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization_28          │ (None, 256)            │         1,024 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_19 (Dropout)            │ (None, 256)            │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_20 (Dense)                │ (None, 128)            │        32,896 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization_29          │ (None, 128)            │           512 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_20 (Dropout)            │ (None, 128)            │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_21 (Dense)                │ (None, 1)              │           129 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 15,145,281 (57.77 MB)

 Trainable params: 428,801 (1.64 MB)

 Non-trainable params: 14,716,480 (56.14 MB)

Epoch 1/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 74s 768ms/step - accuracy: 0.7325 - loss: 0.5423 - precision_10: 0.7392 - recall_10: 0.7314 - val_accuracy: 0.9921 - val_loss: 0.3455 - val_precision_10: 1.0000 - val_recall_10: 0.9839 - learning_rate: 0.0010
Epoch 2/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 205ms/step - accuracy: 0.9933 - loss: 0.0299 - precision_10: 0.9953 - recall_10: 0.9917 - val_accuracy: 1.0000 - val_loss: 0.2247 - val_precision_10: 1.0000 - val_recall_10: 1.0000 - learning_rate: 0.0010
Epoch 3/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 202ms/step - accuracy: 0.9971 - loss: 0.0137 - precision_10: 0.9943 - recall_10: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1748 - val_precision_10: 1.0000 - val_recall_10: 1.0000 - learning_rate: 0.0010
Epoch 4/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 211ms/step - accuracy: 0.9946 - loss: 0.0164 - precision_10: 1.0000 - recall_10: 0.9896 - val_accuracy: 1.0000 - val_loss: 0.1529 - val_precision_10: 1.0000 - val_recall_10: 1.0000 - learning_rate: 0.0010
Epoch 5/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 201ms/step - accuracy: 1.0000 - loss: 0.0072 - precision_10: 1.0000 - recall_10: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1446 - val_precision_10: 1.0000 - val_recall_10: 1.0000 - learning_rate: 0.0010
Epoch 6/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 202ms/step - accuracy: 1.0000 - loss: 0.0075 - precision_10: 1.0000 - recall_10: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.1386 - val_precision_10: 0.9841 - val_recall_10: 1.0000 - learning_rate: 0.0010
Epoch 7/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 202ms/step - accuracy: 1.0000 - loss: 0.0053 - precision_10: 1.0000 - recall_10: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.1189 - val_precision_10: 0.9841 - val_recall_10: 1.0000 - learning_rate: 0.0010
Epoch 8/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 219ms/step - accuracy: 0.9994 - loss: 0.0045 - precision_10: 0.9987 - recall_10: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0997 - val_precision_10: 0.9841 - val_recall_10: 1.0000 - learning_rate: 0.0010
Epoch 9/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 204ms/step - accuracy: 1.0000 - loss: 0.0037 - precision_10: 1.0000 - recall_10: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0891 - val_precision_10: 0.9841 - val_recall_10: 1.0000 - learning_rate: 0.0010
Epoch 10/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 204ms/step - accuracy: 1.0000 - loss: 0.0041 - precision_10: 1.0000 - recall_10: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0890 - val_precision_10: 0.9841 - val_recall_10: 1.0000 - learning_rate: 0.0010
Epoch 11/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 204ms/step - accuracy: 1.0000 - loss: 0.0033 - precision_10: 1.0000 - recall_10: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0707 - val_precision_10: 0.9841 - val_recall_10: 1.0000 - learning_rate: 0.0010
Epoch 12/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 204ms/step - accuracy: 1.0000 - loss: 0.0016 - precision_10: 1.0000 - recall_10: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0490 - val_precision_10: 0.9841 - val_recall_10: 1.0000 - learning_rate: 0.0010
Epoch 13/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 215ms/step - accuracy: 1.0000 - loss: 0.0024 - precision_10: 1.0000 - recall_10: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0352 - val_precision_10: 1.0000 - val_recall_10: 1.0000 - learning_rate: 0.0010
Epoch 14/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 205ms/step - accuracy: 1.0000 - loss: 0.0026 - precision_10: 1.0000 - recall_10: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0261 - val_precision_10: 1.0000 - val_recall_10: 1.0000 - learning_rate: 0.0010
Epoch 15/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 205ms/step - accuracy: 1.0000 - loss: 0.0018 - precision_10: 1.0000 - recall_10: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0210 - val_precision_10: 1.0000 - val_recall_10: 1.0000 - learning_rate: 0.0010
Epoch 1/10
12/12 ━━━━━━━━━━━━━━━━━━━━ 16s 743ms/step - accuracy: 0.9989 - loss: 0.0068 - precision_11: 0.9978 - recall_11: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0403 - val_precision_11: 0.9841 - val_recall_11: 1.0000 - learning_rate: 1.0000e-04
Epoch 2/10
12/12 ━━━━━━━━━━━━━━━━━━━━ 11s 222ms/step - accuracy: 1.0000 - loss: 0.0017 - precision_11: 1.0000 - recall_11: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0316 - val_precision_11: 0.9841 - val_recall_11: 1.0000 - learning_rate: 1.0000e-04
Epoch 3/10
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 217ms/step - accuracy: 1.0000 - loss: 0.0018 - precision_11: 1.0000 - recall_11: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0143 - val_precision_11: 0.9841 - val_recall_11: 1.0000 - learning_rate: 1.0000e-04
Epoch 4/10
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 216ms/step - accuracy: 1.0000 - loss: 0.0014 - precision_11: 1.0000 - recall_11: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0082 - val_precision_11: 1.0000 - val_recall_11: 1.0000 - learning_rate: 1.0000e-04
Epoch 5/10
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 223ms/step - accuracy: 1.0000 - loss: 0.0018 - precision_11: 1.0000 - recall_11: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0102 - val_precision_11: 1.0000 - val_recall_11: 1.0000 - learning_rate: 1.0000e-04
Epoch 6/10
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 222ms/step - accuracy: 1.0000 - loss: 0.0016 - precision_11: 1.0000 - recall_11: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0026 - val_precision_11: 1.0000 - val_recall_11: 1.0000 - learning_rate: 1.0000e-04
Epoch 7/10
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 214ms/step - accuracy: 1.0000 - loss: 9.9746e-04 - precision_11: 1.0000 - recall_11: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0046 - val_precision_11: 1.0000 - val_recall_11: 1.0000 - learning_rate: 1.0000e-04
Epoch 8/10
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 219ms/step - accuracy: 1.0000 - loss: 8.2732e-04 - precision_11: 1.0000 - recall_11: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0041 - val_precision_11: 1.0000 - val_recall_11: 1.0000 - learning_rate: 1.0000e-04
Epoch 9/10
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 218ms/step - accuracy: 1.0000 - loss: 6.6513e-04 - precision_11: 1.0000 - recall_11: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0011 - val_precision_11: 1.0000 - val_recall_11: 1.0000 - learning_rate: 1.0000e-04
Epoch 10/10
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 226ms/step - accuracy: 1.0000 - loss: 6.5313e-04 - precision_11: 1.0000 - recall_11: 1.0000 - val_accuracy: 1.0000 - val_loss: 6.4915e-04 - val_precision_11: 1.0000 - val_recall_11: 1.0000 - learning_rate: 1.0000e-04

==================================================
Model 3 Performance Evaluation
==================================================
12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 209ms/step
4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 302ms/step
Training Performance:
   Accuracy  Recall  Precision  F1 Score
0       1.0     1.0        1.0       1.0

Validation Performance:
   Accuracy  Recall  Precision  F1 Score
0       1.0     1.0        1.0       1.0

Visualizing the predictions¶

In [41]:
visualize_predictions(model3, X_val_rgb, y_val, "VGG16 with FFNN")
plot_confusion_matrix(model3, X_val_rgb, y_val)

1/1 ━━━━━━━━━━━━━━━━━━━━ 2s 2s/step
No description has been provided for this image 
4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 144ms/step
No description has been provided for this image

Model 4: (VGG-16 (Base + FFNN + Data Augmentation)¶

  • In most of the real-world case studies, it is challenging to acquire a large number of images and then train CNNs.

  • To overcome this problem, one approach we might consider is Data Augmentation.

  • CNNs have the property of translational invariance, which means they can recognise an object even if its appearance shifts translationally in some way. - Taking this attribute into account, we can augment the images using the techniques listed below

    • Horizontal Flip (should be set to True/False)
    • Vertical Flip (should be set to True/False)
    • Height Shift (should be between 0 and 1)
    • Width Shift (should be between 0 and 1)
    • Rotation (should be between 0 and 180)
    • Shear (should be between 0 and 1)
    • Zoom (should be between 0 and 1) etc.

Remember, data augmentation should not be used in the validation/test data set.

In [42]:
# Create data augmentation generator
train_datagen = ImageDataGenerator(
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True,
    shear_range=0.15,
    zoom_range=0.15,
    fill_mode='nearest'
)

# Validation data should not be augmented
val_datagen = ImageDataGenerator()

# Prepare data for generators
train_datagen.fit(X_train_rgb)
val_datagen.fit(X_val_rgb)

# Build Model 4 (same architecture as Model 3)
model4 = build_vgg16_ffnn(input_shape=(200, 200, 3))

model4.compile(
    optimizer=Adam(learning_rate=0.001),
    loss='binary_crossentropy',
    metrics=['accuracy', tf.keras.metrics.Precision(), tf.keras.metrics.Recall()]
)

print("Model 4: VGG16 with FFNN + Data Augmentation")
model4.summary()

# Train Model 4 with augmented data
history4 = model4.fit(
    train_datagen.flow(X_train_rgb, y_train, batch_size=32),
    validation_data=val_datagen.flow(X_val_rgb, y_val, batch_size=32),
    epochs=25,
    verbose=1,
    callbacks=[
        tf.keras.callbacks.EarlyStopping(patience=7, restore_best_weights=True),
        tf.keras.callbacks.ReduceLROnPlateau(patience=4, factor=0.5, min_lr=1e-6)
    ]
)

# Fine-tune Model 4
model4.layers[0].trainable = True
for layer in model4.layers[0].layers[:-4]:
    layer.trainable = False

model4.compile(
    optimizer=Adam(learning_rate=0.00005),
    loss='binary_crossentropy',
    metrics=['accuracy', tf.keras.metrics.Precision(), tf.keras.metrics.Recall()]
)

# Continue training with fine-tuning
history4_finetune = model4.fit(
    train_datagen.flow(X_train_rgb, y_train, batch_size=32),
    validation_data=val_datagen.flow(X_val_rgb, y_val, batch_size=32),
    epochs=15,
    verbose=1,
    callbacks=[
        tf.keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True),
        tf.keras.callbacks.ReduceLROnPlateau(patience=3, factor=0.5, min_lr=1e-7)
    ]
)

# Evaluate Model 4
print("\n" + "="*50)
print("Model 4 Performance Evaluation")
print("="*50)
train_perf4 = model_performance_classification(model4, X_train_rgb, y_train)
val_perf4 = model_performance_classification(model4, X_val_rgb, y_val)
print("Training Performance:")
print(train_perf4)
print("\nValidation Performance:")
print(val_perf4)

Model 4: VGG16 with FFNN + Data Augmentation

Model: "sequential_10"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ vgg16 (Functional)              │ (None, 512)            │    14,714,688 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_22 (Dense)                │ (None, 512)            │       262,656 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization_30          │ (None, 512)            │         2,048 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_21 (Dropout)            │ (None, 512)            │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_23 (Dense)                │ (None, 256)            │       131,328 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization_31          │ (None, 256)            │         1,024 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_22 (Dropout)            │ (None, 256)            │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_24 (Dense)                │ (None, 128)            │        32,896 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization_32          │ (None, 128)            │           512 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_23 (Dropout)            │ (None, 128)            │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_25 (Dense)                │ (None, 1)              │           129 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 15,145,281 (57.77 MB)

 Trainable params: 428,801 (1.64 MB)

 Non-trainable params: 14,716,480 (56.14 MB)

Epoch 1/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 17s 858ms/step - accuracy: 0.6895 - loss: 0.5919 - precision_12: 0.7045 - recall_12: 0.6987 - val_accuracy: 0.9921 - val_loss: 0.3375 - val_precision_12: 1.0000 - val_recall_12: 0.9839 - learning_rate: 0.0010
Epoch 2/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 360ms/step - accuracy: 0.9923 - loss: 0.0377 - precision_12: 0.9847 - recall_12: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1960 - val_precision_12: 1.0000 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 3/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 359ms/step - accuracy: 0.9843 - loss: 0.0422 - precision_12: 0.9905 - recall_12: 0.9767 - val_accuracy: 1.0000 - val_loss: 0.1355 - val_precision_12: 1.0000 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 4/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 445ms/step - accuracy: 0.9984 - loss: 0.0187 - precision_12: 0.9969 - recall_12: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1086 - val_precision_12: 1.0000 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 5/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 365ms/step - accuracy: 1.0000 - loss: 0.0103 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0946 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 6/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 361ms/step - accuracy: 1.0000 - loss: 0.0032 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0851 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 7/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 372ms/step - accuracy: 1.0000 - loss: 0.0054 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0751 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 8/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 362ms/step - accuracy: 0.9834 - loss: 0.0525 - precision_12: 1.0000 - recall_12: 0.9668 - val_accuracy: 0.9921 - val_loss: 0.0855 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 9/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 423ms/step - accuracy: 1.0000 - loss: 0.0058 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0755 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 10/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 360ms/step - accuracy: 0.9994 - loss: 0.0046 - precision_12: 0.9987 - recall_12: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0628 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 11/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 357ms/step - accuracy: 1.0000 - loss: 0.0080 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0628 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 12/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 453ms/step - accuracy: 1.0000 - loss: 0.0028 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0525 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 13/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 364ms/step - accuracy: 1.0000 - loss: 0.0028 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0387 - val_precision_12: 1.0000 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 14/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 412ms/step - accuracy: 1.0000 - loss: 0.0080 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0454 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 15/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 6s 460ms/step - accuracy: 1.0000 - loss: 0.0088 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0385 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 16/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 354ms/step - accuracy: 0.9783 - loss: 0.0555 - precision_12: 0.9991 - recall_12: 0.9600 - val_accuracy: 0.9921 - val_loss: 0.0464 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 17/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 355ms/step - accuracy: 1.0000 - loss: 0.0045 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0867 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 18/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 6s 384ms/step - accuracy: 0.9947 - loss: 0.0112 - precision_12: 1.0000 - recall_12: 0.9895 - val_accuracy: 0.9921 - val_loss: 0.0473 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 19/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 361ms/step - accuracy: 0.9888 - loss: 0.0226 - precision_12: 0.9769 - recall_12: 0.9987 - val_accuracy: 0.9921 - val_loss: 0.0382 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 20/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 420ms/step - accuracy: 1.0000 - loss: 0.0076 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0167 - val_precision_12: 1.0000 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 21/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 356ms/step - accuracy: 1.0000 - loss: 0.0033 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0055 - val_precision_12: 1.0000 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 22/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 364ms/step - accuracy: 0.9843 - loss: 0.0295 - precision_12: 0.9689 - recall_12: 0.9942 - val_accuracy: 1.0000 - val_loss: 0.0031 - val_precision_12: 1.0000 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 23/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 457ms/step - accuracy: 0.9980 - loss: 0.0040 - precision_12: 0.9982 - recall_12: 0.9977 - val_accuracy: 1.0000 - val_loss: 0.0021 - val_precision_12: 1.0000 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 24/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 360ms/step - accuracy: 0.9989 - loss: 0.0026 - precision_12: 0.9977 - recall_12: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0039 - val_precision_12: 1.0000 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 25/25
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 356ms/step - accuracy: 0.9973 - loss: 0.0044 - precision_12: 0.9944 - recall_12: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0112 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010
Epoch 1/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 18s 889ms/step - accuracy: 0.9982 - loss: 0.0071 - precision_13: 0.9964 - recall_13: 1.0000 - val_accuracy: 0.9841 - val_loss: 0.0620 - val_precision_13: 0.9688 - val_recall_13: 1.0000 - learning_rate: 5.0000e-05
Epoch 2/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 369ms/step - accuracy: 0.9960 - loss: 0.0078 - precision_13: 0.9987 - recall_13: 0.9934 - val_accuracy: 0.9921 - val_loss: 0.0373 - val_precision_13: 0.9841 - val_recall_13: 1.0000 - learning_rate: 5.0000e-05
Epoch 3/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 372ms/step - accuracy: 1.0000 - loss: 0.0011 - precision_13: 1.0000 - recall_13: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0051 - val_precision_13: 1.0000 - val_recall_13: 1.0000 - learning_rate: 5.0000e-05
Epoch 4/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 426ms/step - accuracy: 0.9777 - loss: 0.0256 - precision_13: 0.9619 - recall_13: 0.9875 - val_accuracy: 1.0000 - val_loss: 0.0057 - val_precision_13: 1.0000 - val_recall_13: 1.0000 - learning_rate: 5.0000e-05
Epoch 5/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 361ms/step - accuracy: 1.0000 - loss: 0.0022 - precision_13: 1.0000 - recall_13: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0145 - val_precision_13: 0.9841 - val_recall_13: 1.0000 - learning_rate: 5.0000e-05
Epoch 6/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 389ms/step - accuracy: 0.9973 - loss: 0.0097 - precision_13: 1.0000 - recall_13: 0.9944 - val_accuracy: 0.9921 - val_loss: 0.0173 - val_precision_13: 0.9841 - val_recall_13: 1.0000 - learning_rate: 5.0000e-05
Epoch 7/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 368ms/step - accuracy: 1.0000 - loss: 0.0044 - precision_13: 1.0000 - recall_13: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0080 - val_precision_13: 0.9841 - val_recall_13: 1.0000 - learning_rate: 2.5000e-05
Epoch 8/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 368ms/step - accuracy: 1.0000 - loss: 0.0020 - precision_13: 1.0000 - recall_13: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0050 - val_precision_13: 1.0000 - val_recall_13: 1.0000 - learning_rate: 2.5000e-05
Epoch 9/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 458ms/step - accuracy: 1.0000 - loss: 0.0011 - precision_13: 1.0000 - recall_13: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0043 - val_precision_13: 1.0000 - val_recall_13: 1.0000 - learning_rate: 2.5000e-05
Epoch 10/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 363ms/step - accuracy: 1.0000 - loss: 0.0050 - precision_13: 1.0000 - recall_13: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0039 - val_precision_13: 1.0000 - val_recall_13: 1.0000 - learning_rate: 2.5000e-05
Epoch 11/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 368ms/step - accuracy: 1.0000 - loss: 0.0027 - precision_13: 1.0000 - recall_13: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0016 - val_precision_13: 1.0000 - val_recall_13: 1.0000 - learning_rate: 2.5000e-05
Epoch 12/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 6s 467ms/step - accuracy: 1.0000 - loss: 0.0047 - precision_13: 1.0000 - recall_13: 1.0000 - val_accuracy: 1.0000 - val_loss: 3.1216e-04 - val_precision_13: 1.0000 - val_recall_13: 1.0000 - learning_rate: 2.5000e-05
Epoch 13/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 374ms/step - accuracy: 1.0000 - loss: 0.0040 - precision_13: 1.0000 - recall_13: 1.0000 - val_accuracy: 1.0000 - val_loss: 2.7637e-04 - val_precision_13: 1.0000 - val_recall_13: 1.0000 - learning_rate: 2.5000e-05
Epoch 14/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 364ms/step - accuracy: 1.0000 - loss: 0.0016 - precision_13: 1.0000 - recall_13: 1.0000 - val_accuracy: 1.0000 - val_loss: 3.0038e-04 - val_precision_13: 1.0000 - val_recall_13: 1.0000 - learning_rate: 2.5000e-05
Epoch 15/15
12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 433ms/step - accuracy: 1.0000 - loss: 6.8510e-04 - precision_13: 1.0000 - recall_13: 1.0000 - val_accuracy: 1.0000 - val_loss: 3.0791e-04 - val_precision_13: 1.0000 - val_recall_13: 1.0000 - learning_rate: 2.5000e-05

==================================================
Model 4 Performance Evaluation
==================================================
12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 217ms/step
4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 305ms/step
Training Performance:
   Accuracy  Recall  Precision  F1 Score
0       1.0     1.0        1.0       1.0

Validation Performance:
   Accuracy  Recall  Precision  F1 Score
0       1.0     1.0        1.0       1.0

Visualizing the predictions¶

In [43]:
visualize_predictions(model4, X_val_rgb, y_val, "VGG16 with Data Augmentation")
plot_confusion_matrix(model4, X_val_rgb, y_val)

1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 948ms/step
No description has been provided for this image 
4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 149ms/step
No description has been provided for this image

Model Performance Comparison and Final Model Selection¶

In [44]:
performance_comparison = pd.DataFrame({
    'Model': ['Simple CNN', 'VGG16 Base', 'VGG16 + FFNN', 'VGG16 + FFNN + Aug'],
    'Val_Accuracy': [
        val_perf1['Accuracy'].values[0],
        val_perf2['Accuracy'].values[0],
        val_perf3['Accuracy'].values[0],
        val_perf4['Accuracy'].values[0]
    ],
    'Val_Precision': [
        val_perf1['Precision'].values[0],
        val_perf2['Precision'].values[0],
        val_perf3['Precision'].values[0],
        val_perf4['Precision'].values[0]
    ],
    'Val_Recall': [
        val_perf1['Recall'].values[0],
        val_perf2['Recall'].values[0],
        val_perf3['Recall'].values[0],
        val_perf4['Recall'].values[0]
    ],
    'Val_F1': [
        val_perf1['F1 Score'].values[0],
        val_perf2['F1 Score'].values[0],
        val_perf3['F1 Score'].values[0],
        val_perf4['F1 Score'].values[0]
    ]
})

print("="*60)
print("MODEL PERFORMANCE COMPARISON (Validation Set)")
print("="*60)
print(performance_comparison.round(4))

# Visualize comparison
fig, axes = plt.subplots(2, 2, figsize=(14, 10))

metrics = ['Val_Accuracy', 'Val_Precision', 'Val_Recall', 'Val_F1']
colors = ['#3498db', '#e74c3c', '#2ecc71', '#f39c12']

for idx, (ax, metric) in enumerate(zip(axes.ravel(), metrics)):
    bars = ax.bar(range(len(performance_comparison)),
                   performance_comparison[metric],
                   color=colors)
    ax.set_xticks(range(len(performance_comparison)))
    ax.set_xticklabels(performance_comparison['Model'], rotation=45, ha='right')
    ax.set_ylabel('Score')
    ax.set_title(metric.replace('Val_', '').replace('_', ' '))
    ax.set_ylim([0, 1])
    ax.grid(axis='y', alpha=0.3)

    # Add value labels on bars
    for bar, value in zip(bars, performance_comparison[metric]):
        ax.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 0.01,
                f'{value:.3f}', ha='center', va='bottom', fontweight='bold')

plt.suptitle('Model Performance Comparison', fontsize=16, fontweight='bold')
plt.tight_layout()
plt.show()

# Select best model based on F1 score
best_model_idx = performance_comparison['Val_F1'].idxmax()
best_model_name = performance_comparison.loc[best_model_idx, 'Model']
best_model = [model1, model2, model3, model4][best_model_idx]

print(f"\n{'='*60}")
print(f"BEST MODEL SELECTED: {best_model_name}")
print(f"{'='*60}")
print(f"Validation F1 Score: {performance_comparison.loc[best_model_idx, 'Val_F1']:.4f}")
print(f"Validation Accuracy: {performance_comparison.loc[best_model_idx, 'Val_Accuracy']:.4f}")

============================================================
MODEL PERFORMANCE COMPARISON (Validation Set)
============================================================
                Model  Val_Accuracy  Val_Precision  Val_Recall  Val_F1
0          Simple CNN        0.5079          0.258      0.5079  0.3422
1          VGG16 Base        1.0000          1.000      1.0000  1.0000
2        VGG16 + FFNN        1.0000          1.000      1.0000  1.0000
3  VGG16 + FFNN + Aug        1.0000          1.000      1.0000  1.0000
No description has been provided for this image 
============================================================
BEST MODEL SELECTED: VGG16 Base
============================================================
Validation F1 Score: 1.0000
Validation Accuracy: 1.0000

Test Performance¶

In [47]:
if best_model_idx == 0:  # Simple CNN uses grayscale
    X_test_final = X_test
else:  # VGG models use RGB
    X_test_final = X_test_rgb

# Evaluate on test set
test_performance = model_performance_classification(
    best_model,
    X_test_final,
    y_test
)

print(f"\nBest Model: {best_model_name}")
print("\nTest Set Performance:")
print(test_performance)

# Detailed classification report
test_predictions = (best_model.predict(X_test_final) > 0.5).astype(int).reshape(-1)
print("\nDetailed Classification Report:")
print(classification_report(y_test, test_predictions,
                          target_names=['Without Helmet', 'With Helmet']))

# Final confusion matrix
print("\nTest Set Confusion Matrix:")
plot_confusion_matrix(best_model,
                     X_test_final,
                     y_test)

# Visualize test predictions
visualize_predictions(best_model, X_test_final, y_test,
                     f"Final Model ({best_model_name}) - Test Set")

# Calculate and display business metrics
tn, fp, fn, tp = confusion_matrix(y_test, test_predictions).ravel()
safety_compliance_rate = tp / (tp + fn)  # Recall for helmet class
false_alarm_rate = fp / (fp + tn)

4/4 ━━━━━━━━━━━━━━━━━━━━ 11s 4s/step

Best Model: VGG16 Base

Test Set Performance:
   Accuracy  Recall  Precision  F1 Score
0       1.0     1.0        1.0       1.0
4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 141ms/step

Detailed Classification Report:
                precision    recall  f1-score   support

Without Helmet       1.00      1.00      1.00        64
   With Helmet       1.00      1.00      1.00        63

      accuracy                           1.00       127
     macro avg       1.00      1.00      1.00       127
  weighted avg       1.00      1.00      1.00       127


Test Set Confusion Matrix:
4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 146ms/step
No description has been provided for this image 
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 107ms/step
No description has been provided for this image

Actionable Insights & Recommendations¶

Critical metric focus¶

  • Insight: Missing a “Without Helmet” case (false negative) is higher risk than a false alert.
  • Recommendation: Optimize threshold for maximum recall on the violation class (sweep 0.30–0.50) while keeping false alarm rate acceptable (<10%). Track Recall(Without Helmet), False Alarm Rate, and F1 weekly.

Best model leverage¶

  • Insight: VGG16 + FFNN + Augmentation generalizes better than simpler or unfine-tuned variants.
  • Recommendation: Promote this model, export with reproducible preprocessing, and lock version. Begin fine-tuning only if new data shifts distributions.

Data-driven robustness¶

  • Insight: Current dataset size is small; likely errors under low light, occlusion, small head size, or unusual helmet colors.
  • Recommendation: Target incremental data collection (≥15% new samples) focusing on failure modes; add brightness/contrast, noise, and slight blur augmentations next cycle.

Deployment & evolution path¶

  • Insight: Current architecture may be heavy for real-time multi-camera scaling and lacks multi-person localization.
  • Recommendation: Plan edge-friendly distilled / MobileNetV3 variant for latency; roadmap Phase 2 to object detection (YOLO/EfficientDet) for per-worker monitoring and richer safety analytics.

Power Ahead!